Picture decoding method for decoding coded picture data and performing distortion removal by comparing pixel difference values with threshold

ABSTRACT

A picture coding method provides coding an input picture. The method includes dividing the input picture into a plurality of blocks, coding the input picture on a block basis to obtain coded data of each block, and decoding the coded data of each block on a block basis to obtain a reconstructed picture having a plurality of blocks. The method also includes removing coding distortion in an area disposed on both sides of a block boundary between a first block and an adjacent second block in the reconstructed picture having the plurality of blocks. Each block is adaptively decoded either as a field structure block, including only even field pixels or only odd field pixels, or a frame structure block, including odd field pixels and even field pixels. The method further includes storing the reconstructed picture, for which coding distortion is removed, as a reference picture.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.16/000,286, filed Jun. 5, 2018, which is a continuation of U.S. patentapplication Ser. No. 15/812,203, filed Nov. 14, 2017, which issued asU.S. Pat. No. 10,015,517 on Jul. 3, 2018, which is a continuation ofU.S. patent application Ser. No. 14/739,541, filed Jun. 15, 2015, whichissued as U.S. Pat. No. 9,900,614 on Feb. 20, 2018, which is acontinuation of U.S. patent application Ser. No. 11/929,416, filed Oct.30, 2007, which is now abandoned, which is a continuation of U.S. patentapplication Ser. No. 11/433,464, filed May 15, 2006, which is nowabandoned, which is a continuation of U.S. patent application Ser. No.10/451,628, filed Jul. 8, 2003, which issued as U.S. Pat. No. 7,095,787on Aug. 22, 2006, which is a U.S. National Stage application of PCTapplication PCT/JP02/12488, filed on Nov. 29, 2002, and claims thebenefit of U.S. Provisional Application No. 60/333,763, filed Nov. 29,2001, U.S. Provisional Application No. 60/333,767, filed Nov. 29, 2001,and U.S. Provisional Application No. 60/394,312, filed Jul. 9, 2002, aswell as Japanese Application Nos. 2002-008859, filed Jan. 17, 2002,2002-110748, filed Apr. 12, 2002, 2002-127101, filed Apr. 26, 2002, and2002-291264, filed Oct. 3, 2002. The disclosure of each of theabove-mentioned applications is expressly incorporated herein byreference in its entirety.

The present invention relates to a coding distortion removal method forremoving coding distortion that occurs when encoding a video signal, anencoding and a decoding method for increasing the compression rate usingthis coding distortion removal method, and a data recording mediumstoring a program for implementing these methods in software.

BACKGROUND ART

Through advances in digital technologies combing multiple audio, video,and other kinds of pixel streams into a single transmission stream,conventional information media, that is means of communicatinginformation to people such as newspapers, magazines, television, radio,and the telephone, can now be used for multimedia communication.“Multimedia” generally refers to text, graphics, audio, and video linkedtogether in a single transmission stream, but conventional informationmedia must first be digitized before the information can be handled in amultimedia format.

The estimated storage capacity needed to store the information carriedby conventional information media when converted to digital data is only1 or 2 bytes per character for text, but 64 kbits for one second oftelephone quality audio, and 100 Mbits for one second of video atcurrent television receiver quality. It is therefore not practical tohandle these massive amounts of video, and other kinds of pixel streamsinto a single transmission stream, conventional information media, thatis, means of communicating information to people such as newspapers,magazines, television, radio, and the telephone, can now be used formultimedia communication. “Multimedia” generally refers to text,graphics, audio, and video linked together in a single transmissionstream, but conventional information media must first be digitizedbefore the information can be handled in a multimedia format.

The estimated storage capacity needed to store the information carriedby conventional information media when converted to digital data is only1 or 2 bytes per character for text, but 64 kbits for one second oftelephone quality audio, and 100 Mbits for one second of video atcurrent television receiver quality. It is therefore not practical tohandle these massive amounts of information in digital form on the aboveinformation media. For example, video telephony service is availableover ISDN (Integrated Services Digital Network) lines with atransmission speed of 64 Kbps to 1.5 Mbps, but television camera gradevideo cannot be sent as is over ISDN lines.

Data compression therefore becomes essential. Video telephony service,for example, is implemented by using video compression techniquesinternationally standardized in ITU-T (International TelecommunicationUnion, Telecommunication Standardization Sector) Recommendations H.261and H.263, Using the data compression methods defined in MPEG-1, videoinformation can be recorded with audio on a conventional audio CD(Compact Disc).

The MPEG (Moving Picture Experts Group) is an international standard fordigitally compressing moving picture signals (video), MPEG-1 enablescompressing a video signal to 1.5 Mbps, that is, compressing theinformation in a television signal approximately 100:1. Furthermore,because the transmission speed for MPEG-1 video is limited toapproximately 1.5 Mbps, MPEG-2, which was standardized to meet thedemand for even higher picture quality, enables compressing a movingpicture signal to 2 Mbps to 15 Mbps.

MPEG-4 with an even higher compression rate has also been standardizedby the working group (ISO/IEC JTC1/SC29/WG11) that has advanced thestandardization of MPEG-1 and MPEG-2, MPEG-4 not only enables low bitrate, high efficiency coding, it also introduces a powerful error isresistance technology capable of reducing subjective image degradationeven when transmission path errors occur. The ITU-T is also working onstandardizing Recommendation H.26L as a next-generation picture codingmethod.

Unlike conventional video coding techniques, H.26L uses a codingdistortion removal method accompanied by complex processing to removecoding distortion. Block unit coding methods using orthogonal transformssuch as the DCT techniques widely used in video coding are known to besubject to a grid-like distortion known as block distortion at thecoding block boundaries. Because image quality loss in low frequencycomponents is more conspicuous than image quality loss in high frequencycomponents, the low frequency components are coded more faithfully thanthe high frequency components in block unit coding. Furthermore, becausenatural images captured with a camera, for example, contain more lowfrequency components than high frequency components, the coding blockscontain more low frequency components than high frequency components.The coding blocks therefore tend to have substantially no high frequencycomponents and adjacent pixels in a block tend to have substantially thesame pixel value.

Furthermore, because coding is by block unit, there is no assurance thatthe pixel values will be substantially the same at the boundary betweenadjacent blocks, that is, that the pixel values will change continuouslyacross the block boundary, even if the pixel values are substantiallyidentical within each block. The result is that, as shown in FIG. 31describing the concept of coding distortion removal, while the change inpixel values is smooth and continuous in the source image across theblock boundary indicated by the dotted line as shown in FIG. 31(a), andthe pixel values change continuously within each block as shown in FIG.31(b) after the source image is coded by block unit, block distortion,that is, a discontinuity in pixel values only at the block boundary,occurs. Block distortion is thus a significant image quality problemresulting from image coding, but can be reduced by correcting the pixelvalues to be continuous across the block boundary as shown in FIG.31(c). This process of reducing block distortion is called codingdistortion removal (also referred to as “deblocking”).

When deblocking is applied at the video decoding stage, the deblockingfilter can be used as a post filter as shown in the block diagram of avideo decoder using a conventional decoding method in FIG. 32, or it canbe used as an in-loop filter as shown in the block diagram of a videodecoder using a conventional decoding method in FIG. 33. Theconfigurations shown in these block diagrams are described below.

In the block diagram of a video decoder using a conventional decodingmethod shown in FIG. 32, a variable length decoder 52 variable lengthdecodes encoded signal Str and outputs frequency code component DCoef. Ade-zigzag scanning unit 54 rearranges the frequency components of thefrequency code component DCoef in two-dimensional blocks, and outputsfrequency component FCoef, the block unit frequency components. Thereverse cosine transform unit 56 applies dequantization and reverse DCToperations to frequency component FCoef, and outputs difference imageDifCoef.

Motion compensator 60 outputs the pixel at the position indicated byexternally input motion vector MV from the reference image Refaccumulated in memory 64 as motion compensated image MCpel. Adder 58adds difference image DifCoef and motion compensated image MCpel tooutput reconstructed image Coef. Deblocking filter 62 applies codingdistortion removal to reconstructed image Coef, and outputs decodedimage signal Vout. Reconstructed image Coef is stored in memory 64, andused as reference image Ref for the next image decoding.

The block diagram in FIG. 33 of a video decoder using a conventionaldecoding method is substantially identical to the block diagram of avideo decoder shown in FIG. 32, but differs in the location of thedeblocking filter 62. As will be known from FIG. 33 the decoded imagesignal Vout output from deblocking filter 62 is stored to memory 64.

The block diagram in FIG. 32 of a video decoder using a conventionaldecoding method shows the configuration and method used in MPEG-1,MPEG-2, MPEG-4, and H.263. The block diagram in FIG. 33 of a videodecoder using a conventional decoding method shows the configuration andmethod used in H.261 and H.26L TM8.

With the block diagram in FIG. 32 of a video decoder using aconventional decoding method the reconstructed image Coef stored tomemory 64 is not dependent upon the method applied by the deblockingfilter 62. This allows developing and implementing various kinds ofdeblocking filters 62, including complex yet high performance filters aswell as simple filters with relatively little effect according to theperformance of the available hardware and the specific application. Theadvantage is that a deblocking filter 62 appropriate to the device canbe used.

With the block diagram in FIG. 33 of a video decoder using aconventional decoding method the decoded image signal Vout stored tomemory 64 is dependent upon the method employed by the deblocking filter62. The problem here is that the filter cannot be changed to oneappropriate to the hardware or application, but the advantage is thatthe same level of coding distortion removal can be assured in everydevice.

FIG. 34 is a block diagram of a coding distortion removal unit using theconventional coding distortion removal method. FIG. 34 shows theconfiguration of the deblocking filter 62 in FIG. 32 and FIG. 33 indetail. To efficiently remove only coding distortion from an imagesignal containing coding distortion, it is important to determine theamount and tendency for coding distortion in the image signal and thenapply appropriate filtering so as to not degrade the actual imagesignal.

Because high frequency components account for much of the codingdistortion, the general concept behind coding distortion removal is tosurvey the image signal to determine the ratio of high frequencycomponents in the image signal, identify high frequency components inimage signal pixels normally thought to not contain a high frequencycomponent as coding distortion, and apply a high frequency componentsuppression filter to the coding distortion. This is possible becausethe correlation between adjacent pixels in an image signal is high,pixels containing a high frequency component are concentrated in edgeareas, and dispersed high frequency components can be considered to becoding distortion.

This deblocking filter 62 was created by the inventors of the presentinvention based on content found in ITU-T Recommendation H.26L TML8.

Filtered pixel count controller 84 uses reconstructed image Coef todetermine the pixel positions containing coding distortion, and outputsfiltered pixel count FtrPel. Filter coefficient controller 86 usesfiltered pixel count FtrPel and reconstructed image Coef to determinethe filter coefficient (including the number of filter taps) appropriateto removing coding distortion from the indicated pixels, and outputsfilter coefficient FtrTap. The filter processor 88 applies filtering toremove coding distortion from reconstructed image Coef using the filtercoefficient indicated by filter coefficient FtrTap, and outputs decodedimage signal Vout.

DISCLOSURE OF INVENTION

The conventional coding distortion removal methods described above areparticularly effective at removing coding distortion, but the process isextremely complex and implementation difficult.

A further problem is that the amount of data processed per unit time ishigh.

Furthermore, no matter how effective the coding distortion removalmethod, it is impossible to accurately distinguish image signals andcoding distortion without other additional information, and there is,therefore, the possibility that coding distortion removal will degradeimage quality. This problem is particularly great with a configurationas shown in the block diagram in FIG. 33 of a video decoder using aconventional decoding method because the result of deblocking is used asthe reference image and therefore affects the result of coding eachsubsequent picture.

An object of the present invention is therefore to provide a simplecoding distortion removal method.

A further object is to provide a coding distortion removal method, acoding method, and a decoding method whereby the likelihood of degradingimage signal quality can be reduced by applying high performance codingdistortion removal with less possibility of degrading image signalquality as a result of removing coding distortion than the prior art.

To achieve this object, a coding distortion removal method according tothe present invention for removing coding distortion from a picture usesdifferent methods to remove coding distortion at boundaries where themotion compensation unit boundary matches the coding unit boundarymatch, and boundary depending on whether the boundary is a motioncompensation block boundary or not, when the motion compensation blocksize is larger than the coding block size.

Because coding distortion at the boundary of the motion compensationunit differs qualitatively from coding distortion at the coding unitboundary, coding distortion can be efficiently removed from an imagesignal containing coding distortion by changing the filter used fordeblocking according to the unit.

Furthermore, when coded motion compensation error is 0, codingdistortion is preferably removed only at the motion compensation blockboundary.

A further aspect of the invention is a coding distortion removal methodfor removing coding distortion from a picture by means of a step forextracting picture parameters from a picture containing codingdistortion; a first step for identifying pixels for coding distortionremoval using the picture parameters; a second step for identifying themethod for coding distortion removal using the picture parameters; and athird step for removing coding distortion from the pixels identified bythe first step using the coding distortion removal method identified bythe second step.

By first computing picture parameters that can be used in both the firststep identifying the pixels from which coding distortion is removed andthe second step identifying the method used to remove the codingdistortion, the operations performed in the first step and second stepcan be simplified by using these common picture parameters, andprocessing by the coding distortion removal method can be reducedwithout degrading image quality.

A further aspect of the invention is a coding distortion removal methodfor removing coding distortion from a picture whereby the pixels to beprocessed for coding distortion removal are identified by block baseddetermination whether to remove coding distortion by block unit, andthen pixel based determination whether to remove coding distortion foreach pixel in the blocks determined to be removed by the block baseddetermination.

By thus first determining by block unit whether coding distortionremoval is needed, evaluation by pixel unit can be omitted in thoseblocks that do not need deblocking, and the processing performed by thecoding distortion removal method can be reduced. Blocks that do not needdeblocking (such as still image blocks where the pixels perfectly matchthe reference image) can be easily determined if the image codinginformation is used.

A yet further aspect of the invention is a coding distortion removalmethod for removing coding distortion in an area disposed on both sidesof a block boundary between a first block and an adjacent second blockin a picture having a plurality of blocks forming a moving pictureimage. This method has a comparison step for comparing a difference ofpixel values of the first block and pixel values in pixels of the secondblock, and a parameter, corresponding to the average of a quantizationparameter for the first block and a quantization parameter for thesecond block, for determining the method for removing coding distortion;and a removal step for removing coding distortion based on the resultfrom the comparison step.

This enables the average of the quantization parameters for the adjacentblocks to be used when filtering both sides of the block boundary in acoding distortion removal process at the block boundary betweendifferent quantization parameters.

Another coding distortion removal method for removing coding distortionin an area disposed on both sides of a boundary line between a firstblock and an adjacent second block in a picture having a plurality ofblocks forming a moving picture image has a decoding step for decoding aparameter for setting a threshold value when removing coding distortion;a comparison step for comparing a difference of pixel values in pixelsof the first block and pixel values in pixels of the second block, and aspecific threshold value based on the decoded parameter; and a removalstep for switching the method for removing coding distortion based onthe result from the comparison step.

Coding distortion can thus be efficiently removed from an image signalcontaining coding distortion by first superposing to each encoded signala threshold value parameter used for coding distortion removal, and thenprior to coding distortion removal detecting the threshold valueappropriate to each encoded signal and using it to remove codingdistortion.

Further preferably, the moving picture contains a slice composed ofplural blocks; and the parameter is stored in slice header informationin a code stream obtained by encoding image-data for the moving picture.

A further aspect of the invention is a moving picture coding apparatusfor picture coding with reference to at least one of multiple referenceimages, wherein a plurality of coded images obtained by removing codingdistortion using plural methods are the reference images.

By thus using plural images deblocked by at least two methods asreference images and sequentially selecting the appropriate one forreference, the picture obtained by efficiently removing codingdistortion from an image signal containing coding distortion can be usedas the reference image, and the compression rate of moving picturecoding can be increased.

Further preferably the first method of the plural methods is a methodthat does not remove coding distortion in the coded picture, and thesecond method is a method that removes coding distortion in the codedpicture.

A further aspect of the invention is a moving picture decoding apparatusfor decoding with reference to at least one of multiple referenceimages, wherein a plurality of decoded images obtained by removingcoding distortion using plural methods are the reference images.

By thus using plural images deblocked by at least two methods asreference images and sequentially selecting the appropriate one forreference, the picture obtained by efficiently removing codingdistortion from an image signal containing coding distortion can be usedas the reference image, and the coded signal can be corrected decoded,

Further preferably, the first method of the plural methods is a methodthat does not remove coding distortion in the decoded picture, and thesecond method is a method that removes coding distortion in the decodedpicture.

A further aspect of the invention is a coding distortion removal methodfor removing coding distortion in an interlaced picture composed ofodd-fine pixels and even-line pixels. This method has an evaluation stepfor determining if a picture is a picture containing frame structureblocks having a specific number of odd-fine pixels and a specific numberof even-line pixels, a picture containing blocks of one field structurecomposed of a specific number of odd-line pixels, or a picturecontaining blocks of another field structure composed of a specificnumber of even-line pixels; and a removal step for removing codingdistortion between adjacent frame structure blocks when the target blockfor coding distortion removal is a block in a picture in which allblocks are frame structure blocks, and removing coding distortionbetween adjacent field structure blocks when the target block for codingdistortion removal is a block in a picture in which all blocks are fieldstructure blocks.

Processing of the blocks for coding distortion removal can thus bechanged based on whether the blocks are in a picture of frame structureblocks or a picture of field structure blocks.

Preferably, if the target block for coding distortion removal is a blockof a picture containing frame structure blocks and field structureblocks, the coding distortion removal method also has a conversion stepfor converting a field structure block to a frame structure block; acomparison step for comparing a difference of pixel values in pixels ofthe field structure block and pixel values in pixels of the convertedblock with a specific threshold value; and a removal step for removingcoding distortion based on the result from the comparison step.

In a further coding distortion removal method for removing codingdistortion in an area disposed on both sides of a boundary line betweena first block and an adjacent second block in a picture having aplurality of blocks forming a moving picture image, the first blocks areframe structure blocks having a specific number of odd-line pixels and aspecific number of even-line pixels in an interlaced picture composed ofodd-line pixels and even-line pixels, and the second blocks are fieldstructure blocks having one field composed of a specific number ofodd-line pixels in an interlaced picture composed of odd-line pixels andeven-line pixels, and another field composed of a specific number ofeven-line pixels in an interlaced picture composed of odd-line pixelsand even-line pixels. The coding distortion removal method has aconversion step for converting a frame structure first block to a fieldstructure block; a comparison step for comparing a difference of pixelvalues in pixels of the field structure second block and pixel values inpixels of the converted block with a specific threshold value; and aremoval step for removing coding distortion based on the result from thecomparison step.

When field structure blocks and frame structure blocks are adjacent, thetarget blocks for coding distortion removal can thus be adaptivelyprocessed.

Preferably, conversion from frame structure first blocks to fieldstructure blocks switches by macroblock unit or units of two verticallyadjacent macroblocks.

Further preferably, field structure second blocks are not converted toframe structure blocks.

In a further coding distortion removal method for removing codingdistortion in an area disposed on both sides of a boundary line betweena first block and an adjacent second block in a picture having aplurality of blocks forming a moving picture image, the first blocks areframe structure blocks having a specific number of odd--line pixels anda specific number of even-line pixels in an interlaced picture composedof odd-line pixels and even-line pixels, and the second blocks are fieldstructure blocks having one field composed of a specific number ofodd-line pixels in an interlaced picture composed of odd-line pixels andeven-line pixels, and another field composed of a specific number ofeven-line pixels in an interlaced picture composed of odd-line pixelsand even-line pixels. The coding distortion removal method has anevaluation step for determining if the target block for codingdistortion removal is a frame structure block or a field structureblock; a conversion step for converting the frame structure first blockto a field structure block when the target block is a field structuresecond block, and converting the field structure second block to a framestructure block when the target block is a frame structure first block;a comparison step for comparing pixel values in pixels of the targetblock with a specific threshold value; and a removal step for removingcoding distortion based on the result from the comparison step.

When field structure blocks and frame structure blocks are adjacent, thetarget blocks for coding distortion removal can thus be adaptivelyprocessed.

Preferably, conversion in the conversion step from a frame structureblock to a field structure block produces one field after conversionfrom odd-line pixels in the frame structure block, and produces theother field after conversion from even-line pixels in the framestructure block; and comparison of the difference and threshold value inthe comparison step compares pixel values in pixels in one field of thesecond block and pixel values in pixels in one field of the first blockafter conversion, or compares pixel values in pixels of the other fieldin the second block and pixel values in pixels of the other field in thefirst block after conversion.

In a further coding distortion removal method for removing codingdistortion in an area disposed on both sides of a boundary line betweena first block and an adjacent second block in a picture having aplurality of blocks forming a moving picture image, the first blocks areframe structure blocks having a specific number of odd-line pixels and aspecific number of even-line pixels in an interlaced picture composed ofodd-line pixels and even-line pixels, and the second blocks are fieldstructure blocks having one field composed of a specific number ofodd-line pixels in an interlaced picture composed of odd-line pixels andeven-line pixels, and another field composed of a specific number ofeven-line pixels in an interlaced picture composed of odd-line pixelsand even-line pixels. The coding distortion removal method has aconversion step for converting a field structure second block to a framestructure block; a comparison step for comparing a difference of pixelvalues in pixels of the frame structure first block and pixel values inpixels of the converted block with a specific threshold value; and aremoval step for removing coding distortion based on the result from thecomparison step.

When field structure blocks and frame structure blocks are adjacent, thetarget blocks for coding distortion removal can thus be adaptivelyprocessed.

Further preferably, conversion from field structure second blocks toframe structure blocks switches by macroblock unit or units of twovertically adjacent macroblocks.

Yet further preferably, field structure second blocks are not convertedto frame structure blocks.

Yet further preferably, conversion in the conversion step from fieldstructure block to frame structure block produces a converted frame frompixels in a block of one field and pixels in a block of the other field,and compares pixel values in odd-line pixels in the first block withpixel values in odd-line pixels in the second block after conversion, orcompares pixel values in even-line pixels in the first block with pixelvalues in even-line pixels in the second block after conversion.

Yet further preferably, the comparison step compares the difference andthreshold value by groups of plural pixels aligned in line in a samedirection as the boundary line at positions symmetrical to the boundaryline.

This enables coding distortion to be removed in groups of plural pixels.

A yet further aspect of the present invention is a picture codingapparatus having a decoding unit for decoding a coded difference pictureand outputting the difference picture; a motion compensation unit foroutputting a motion compensated picture from a reference image; an adderfor adding the difference picture and motion compensated picture, andoutputting the merged picture; a coding distortion removal unit forremoving coding distortion in the merged picture and outputting areconstructed picture; and memory for storing the reconstructed pictureas the reference image. The coding distortion removal unit removescoding distortion by means of any of the above-described methods of theinvention.

A yet further aspect of the invention is a program for removing codingdistortion from a picture by means of any of the above-described methodsof the invention.

A yet further aspect of the invention is a program for picture codingusing a decoding unit for decoding a coded difference picture andoutputting the difference picture; a motion compensation unit foroutputting a motion compensated picture from a reference image; an adderfor adding the difference picture and motion compensated picture, andoutputting the merged picture; a coding distortion removal unit forremoving coding distortion in the merged picture and outputting areconstructed picture; and memory for storing the reconstructed pictureas the reference image. The coding distortion removal unit removescoding distortion by means of any of the above-described methods of theinvention.

Other objects and attainments together with a fuller understanding ofthe invention will become apparent and appreciated by referring to thefollowing description and claims taken in conjunction with theaccompanying drawings.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram of a video decoding apparatus using a decodingmethod according to the present invention;

FIG. 2 is a block diagram of a coding distortion removal unit using acoding distortion removal method according to a first embodiment of thepresent invention;

FIGS. 3(a), 3(b), 3(c), 3(d), 3(e), 3(f) and 3(g) show an example of themotion compensation block size;

FIG. 4 is a flow chart of a coding distortion removal method accordingto a second embodiment of the present invention;

FIG. 5 shows the correlation between quantization parameter QP and thecoding distortion removal parameters in a second embodiment of thepresent invention;

FIG. 6 is a flow chart for determining the number of pixels to filter ina coding distortion removal method according to a second embodiment ofthe present invention;

FIG. 7 is a flow chart for determining the filter coefficient in acoding distortion removal method according to a second embodiment of thepresent invention;

FIGS. 8(a) and 8(b) are a block diagram of a coding distortion removalunit using the coding distortion removal method according to secondembodiment of the present invention, and a diagram showing pixelalignment;

FIG. 9 is a block diagram of a coding device using a coding methodaccording to a third embodiment of the present invention;

FIG. 10 is a block diagram of a decoding device using a decoding methodaccording to a third embodiment of the present invention;

FIG. 11 is a block diagram of a coding distortion removal unit using thecoding distortion removal method according to a fourth embodiment of thepresent invention;

FIGS. 12(a), 12(b), 12(c) and 12(d) show the structure of the encodedsignal Str in a coding distortion removal method according to a fourthembodiment of the present invention;

FIG. 13 is a block diagram showing a video encoding process using a loopfilter;

FIG. 14 is a block diagram showing the location of the automaticthreshold value selection in a video encoding loop;

FIG. 15 is a flow chart showing a method for gathering data for findingan optimum threshold value;

FIG. 16 is a flow chart showing another method for gathering data forfinding an optimum threshold value;

FIG. 17 is a flow chart showing a method for selecting an optimizedthreshold value;

FIG. 18 shows the neighborhood of blocks having common boundaries forwhich deblocking can be skipped;

FIG. 19 shows a group containing multiple pixels;

FIG. 20(a) describes a frame structure and FIG. 20(b) describes a fieldstructure;

FIG. 21(a) describes a structure where a frame structure and a fieldstructure are mixed in a single picture, and FIG. 21(b) and FIG. 21(c)describe steps in the coding distortion removal process at the boundarybetween a field structure and frame structure;

FIG. 22 is a flow chart of a coding distortion removal process used whenframe and field structures are mixed;

FIG. 23 is a flow chart for a variation in which steps memory 64 and 67in FIG. 22 are combined;

FIG. 24 is a flow chart for a variation in which steps memory 65 and 68in FIG. 23 are combined;

FIG. 25 is a flow chart of a process used when a frame structure blockand a field structure block are on opposite sides of the block boundary;

FIGS. 26(a), 26(b) and 26(c) describe a recording medium according to asixth embodiment of the present invention for storing acomputer-executable program implementing the variable length coding andvariable length decoding methods of the first and second embodiments ofthe invention;

FIG. 27 is a block diagram showing the overall configuration of acontent supply system;

FIG. 28 shows an exemplary cell phone using a video encoding method andvideo decoding method;

FIG. 29 is a block diagram of a cell phone;

FIG. 30 shows an example of a digital broadcasting system;

FIGS. 31(a), 31(b) and 31(c) show pixel signal level diagrams todescribe the concept of a coding distortion removal method;

FIG. 32 is a block diagram of a video decoding apparatus using adecoding method of the prior art;

FIG. 33 is a block diagram of a video decoding apparatus using adecoding method of the prior art; and

FIG. 34 is a block diagram of a coding distortion removal unit using acoding distortion removal method according to the prior art.

BEST MODE FOR CARRYING OUT THE INVENTION

Preferred embodiments of the present invention are described below withreference to the accompanying figures.

Embodiment 1

In the block diagram of a video decoding apparatus using a videodecoding method, variable length decoder 52 variable length decodesencoded signal Str and outputs frequency code component DCoef. De-zigzagscanning unit 54 rearranges the frequency components of the frequencycode component DCoef in two-dimensional blocks, and outputs frequencycomponent FCoef, the block unit frequency components. The reverse cosinetransform unit 56 applies dequantization and reverse DCT operations tofrequency component FCoef, and outputs difference image DifCoef.

Motion compensator 60 outputs the pixel at the position indicated byexternally input motion vector MV from the reference image Refaccumulated in memory 64 as motion compensated image MCpel, and outputsmotion compensation block size MCsize denoting the size of the motioncompensation block. Adder 58 adds difference image DifCoef and motioncompensated image MCpel to output reconstructed image Coef.

Deblockng filter 62 receives reconstructed image Coef, motioncompensation block size MCsize, and difference image DifCoef, appliescoding distortion removal, and outputs decoded image signal Vout.Reconstructed image Coef is stored in memory 64, and used as referenceimage Ref for the next image decoding.

FIG. 2 is a block diagram of deblocking filter 62 (also called a codingdistortion removal unit) using a coding distortion removal methodaccording to the present invention. This deblocking filter 62 wascreated by the inventors of the present invention with reference to thecontent of a deblocking filter described in ITU-T Recommendation H.26LTML8.

Filtered pixel count controller 4 determines the pixel positionscontaining coding distortion for each reconstructed image Coef, andoutputs filtered pixel count FtrPel. Filtered pixel count FtrPel thusindicates the pixel position that needs filtering.

Filter coefficient controller 6 uses filtered pixel count FtrPel andreconstructed image Coef to determine the filter coefficient (includingthe number of filter taps) appropriate to removing coding distortionfrom the indicated pixels, and outputs filter coefficient FtrTap.

The filter processor 8 applies a filter process to remove codingdistortion from reconstructed image Coef using the filter coefficientindicated by filter coefficient FtrTap, and outputs decoded image signalVout.

The difference image DifCoef and motion compensation block size MCsizeare input to motion compensation block boundary detection unit 2, whichdetermines whether the difference image DifCoef for the process block isless than or equal to a specific value, such as whether it is 0, detectsthe boundaries of the motion compensation block, and outputs motioncompensation block boundary flag is Edge.

FIG. 3 shows examples of the motion compensation block size used inITU-T Recommendation H.26L TML8. As shown in these examples the maximummotion compensation block size is 16×16 pixels, the same size as what isreferred to as a macroblock. The motion compensation block sizes shownin FIG. 3(a) to (g) are 4×4, 4×8, 8×4, 8×8, 8×16, 16×8, and 16×16pixels. In lTU-T Recommendation H.26L TML8 the size appropriate to themacroblock unit is selected from these seven motion compensation blocksizes and used for coding and decoding. It should be noted that codingand decoding can be applied to an appropriate unit of two verticallyadjacent macroblocks, and a unit of such macroblocks is called a“macroblock pair.”

The unit used for frequency transforms and coding in. ITU-TRecommendation H.26L TML8 is 4×4 pixels. This unit of 4×4 pixels iscalled a “coding unit.” As shown in FIG. 3(a), each of the sixteenblocks A to P is a 4×4 pixel block. The 4×4 pixel coding unit matchesthe motion compensation block size only in the case shown in FIG. 3(a).Because block distortion that is particularly visually disruptive ascoding distortion occurs at the smallest coding unit size of 4×4 pixels,the conventional coding distortion removal method always works on 4×4pixel units.

If the correlation between pictures is particularly strong after motioncompensation coding, the coded motion compensation error betweenpictures is 0. Because the difference image DifCoef coded and decoded in4×4 pixel units is also 0 in this case, discontinuities in the pixelvalues resulting from coding distortion during coding and decodinglikely does not occur in places other than the boundaries of the motioncompensation blocks. Therefore, if the motion compensation blocks areselected as shown in FIG. 3(b), the coding distortion removal process isnot needed at the 4×4 pixel unit boundaries indicated by the dottedlines between blocks AC, BD, EG, FH, IK, JL, MO, and NP shown in FIG.3(a). Deblocking is likewise not needed at the 4×4 pixel unit boundariesindicated by the dotted lines between blocks AB, CD, EF, GH, IJ, KL, MN,and OP shown in FIG. 3(a). If difference image DifCoef used forcoding/decoding in 4×4 pixel units is also 0, deblocking is applied onlyat the boundaries of the motion compensation blocks, and is not appliedat the boundaries of the 4×4 pixel units within the motion compensationblocks. This makes it possible to reduce the number of operations in thecoding distortion removal process compared with deblocking all blockboundaries.

If the difference image DifCoef of the process block is 0 and is not asthe boundary of a motion compensation block, motion compensation blockboundary detection unit 2 sets both selectors 10 a and 10 b off(indicated by a solid line) and selector 10 b outputs reconstructedimage Coef as decoded image signal Vout. The selectors 10 a and 10 b areswitched by setting the motion compensation block boundary flag IsEdge.Processing by filtered pixel count controller 4, filter coefficientcontroller 6, and filter processor 8 can thus be omitted by switchingselectors 10 a and 10 b off. In cases other than above, selectors 10 aand 10 b are ON (denoted by the dotted line), and the output from filterprocessor 8 is output from selector 10 b as decoded image signal Vout.This selector state is also set by applying motion compensation blockboundary flag IsEdge.

The present invention thus introduces the ability to omit operation offiltered pixel count controller 4, filter coefficient controller 6, andfilter processor 8 by applying an appropriately set motion compensationblock boundary flag IsEdge, and by skipping these units enables fasterprocessing and reduces power consumption by these processes.

It should be noted that this embodiment is described as simply notapplying any coding distortion removal process, a simple codingdistortion removal process could be used instead of skipping the processaltogether, and switching could be between a complex coding distortionremoval process and coding distortion removal processing in 4×4 pixelunits.

Embodiment 2

A specific process whereby coding distortion removal can be easilyachieved is described in this embodiment of the invention with referenceto the flow chart in FIG. 4 of a coding distortion removal methodaccording to the present invention.

It is first determined in step S18 whether the target block is a codingdistortion removal block. If it is, control advances to step S19. If itis not, control advances to step S24.

An appropriate coding distortion removal filter is selected in step S19,coding distortion removal processing is applied using the selectedfilter in step S20, and the target pixel is changed to the nextunprocessed pixel in the block in step S21. If there are no unprocessedpixels in the block (step S22 returns no), control advances to step S24.If there is an unprocessed pixel (step S22 returns yes), control loopsback to step S19 and the process repeats.

Step S24 detects if there is another unprocessed block in the picture.If there is, control advances to step S23. If all blocks have beenprocessed (step S24 returns no), the coding distortion removal processends for that picture.

If unprocessed blocks remain, the target block is changed to the nextunprocessed block in step S23, control loops back to step S18 and theprocess repeats.

FIG. 6 is a flow chart showing how the number of pixels to filter (the“filtered pixel count” below) is determined in the coding distortionremoval method of the present invention. This flow chart describes oneexample the filtered pixel count controller 4 shown in FIG. 2 couldoperate. FIG. 6 shows a case in which the motion compensation block isthe one shown in FIG. 8(a). As shown in FIG. 8(b), the target pixelvalues for coding distortion removal are

p3, p2, p1, p0, Q0, Q1, Q2, Q3 as shown in FIG. 8(b), and the pixelvalues after coding distortion removal are

P3, P2, P1, P0, Q0, Q1, Q2, Q3. These pixel values are assignedsequentially in the same order as the pixel positions, p0 to p3 and P0to P3 denote corresponding pixels in the same block, and q0 to q3 and Q0to Q3 denote corresponding pixels in the same block.

As quantization parameter QP increases the quantization steps get larger(coarser) and the size of the coding distortion also increases. It istherefore effective to change the filter according to the size ofquantization parameter QP. FIG. 5 is a table showing the correlationbetween quantization parameter QP and coding distortion removalparameters. The correlation between parameters π, Ω, and n of thedeblocking process for determining parameter n denoting the filteredpixel count is shown in Table 1 below. It should be noted that filteringshould not be applied if the pixel difference is large because thisdenotes an edge, and π is therefore preferably set so that filtering isnot applied to pixels where the pixel difference is less than π.Furthermore, if the pixel difference is small the likelihood that thepixels are not at an edge increases as the pixel difference decreases,and Ω is therefore preferably set so that a stronger filter (i.e., n ishigh) is applied based on whether the pixel difference is extremely low(less than Ω) or somewhat small (less than 2×Ω).

TABLE 1 Condition A Condition B n difla > π dif2a < Ω 0 difla > π Ω ≤dif2a ≤ 2 × Ω 0 difla > π dif2a ≥ 2 × Ω 0 difla ≤ π dif2a < Ω 2 difla ≤π Ω ≤ dif2a ≤ 2 × Ω 1 difla ≤ π dif2a ≥ 2 × Ω 0 where dif1 = p0 − q0dif2 = p1 − q1 dif1a = |dif1| dif2a = |dif2|.In other words, the flow chart for determining the filtered pixel countin the coding distortion removal method of the present invention issummarized in Table 1.

Step S27 computes pixel difference DifPel, a parameter that isrepeatedly computed in the coding distortion removal process. Note thatpixel difference DifPel refers to dif1a and dif2a calculated in stepS27.

Step S28 then compares dif1a and π. If dif1a is greater than π, step S29sets n=0 and the process ends without running the coding distortionremoval process. If dif1a is less than or equal to π, control advancesto step S30.

In step S30 dif2a is compared with Ω. If dif2a is less than Ω, step S31sets n=2 (that is, coding distortion removal is applied to the secondpixel to from the boundary of each adjacent block), and the processends. If dif2a is greater than or equal to Ω, control advances to stepS32.

In step S32 dif2a is compared with 2×Ω. If dif2a is less than 2×Ω, stepS33 sets n=1 (that is, coding distortion removal is applied to the firstpixel from the boundary of each adjacent block), and the process ends.If dif2 is the absolute value of the difference in pixel values inproximity to the boundary, and because the number of high frequencycomponents near the boundary decreases as this difference decreases,coding distortion can be removed efficiently from the boundary area byincreasing the number of pixels processed for deblocking as dif2 getssmaller.

FIG. 7 is a flow chart of a process for determining the filtercoefficient in the coding distortion removal method of the presentinvention, and is an example of the operation of filter coefficientcontroller 6 in FIG. 2.

Three conditions are compared using n, dif1a, dif2a, and ø in step S37.If all three conditions are true, a three tap filter process is set instep S39. That is, ø is the threshold value for determining the numberof filter taps, and a three tap filter is applied when the highfrequency component is low (n=2) and there is little change in pixelvalues at the boundary (|dif2a-dif1a|<ø). A three tap filter normallyprovides stronger suppression of high frequency components than a singletap filter. Because the filter process can be changed using the value ofn, parameter n can be used to change the type of filter instead of thenumber of pixels the filter is applied to. Parameter n thus obtained canalso be used to change both the number of pixels filtered and the typeof filter applied.

If the three conditions are not true in step S37, the value of n isdetected in step S38. If n≥1, step S40 sets a one tap filter process. Ifn=0, step S42 turns filtering off.

It should be noted that quantization parameter QP can be changed foreach block. However, the coding distortion removal process becomes morecomplicated at the boundary between blocks having a differentquantization parameter QP. The present invention prevents this by using:

the average quantization parameter QP of adjacent blocks (fractions maybe rounded),

the highest quantization parameter QP of the adjacent blocks,

the lowest quantization parameter QP of the adjacent blocks, or

the quantization parameter QP of the left-adjacent or above-adjacentblock

as the quantization parameter QP for filtering blocks on both sides ofthe boundary when the quantization parameter QP changes in the boundaryblocks. It should be noted that the difference between using these fourquantization parameters QP is little, and one could be preselected foruse.

Coding distortion can thus be easily removed by the method describedabove.

FIG. 8(a) is a block diagram of another embodiment of the deblockingfilter 62 shown in FIG. 1, and a separate embodiment of the partenclosed in a dotted line in FIG. 2. It should be noted that like partsin FIG. 8 and the block diagram of the coding distortion removal unitusing the conventional coding distortion removal method shown in FIG. 34are identified by like reference numerals, and further descriptionthereof is omitted here.

The pixel difference calculator 20 computes the pixel difference at theblock boundary from reconstructed image Coef, and outputs pixeldifference DifPel. This pixel difference DifPel contains a signalequivalent to dif1a and dif2a. Pixel difference DifPel is obtained bycomparing pixels at symmetrical positions left and right or above andbelow the boundary between coding unit blocks, and using the differenced1, d2, d3, d4 (color difference or luminance difference) therebetween.If the average of these differences (e,g., (d1+d2+d3+d4)/4) is less thanor equal to a specific value, an image boundary line is likely notpresent in the range of the width used to determine d4, and thedeblocking filter is therefore applied. On the other hand, if theaverage is greater than or equal to a specific value, there is an imageboundary and the deblocking filter is not applied. It should be notedthat this comparison could use any one, any two, or any three of d1, d2,d3, and d4. Rather than using the average, the highest difference couldalternatively be compared with a specific value.

The flow chart for determining the filtered pixel count can be used asan example of filtered pixel count controller 4 operation. An example offilter coefficient controller 6 operation in this embodiment is shown inthe flow chart for determining the filter coefficient shown in FIG. 7.By referencing pixel difference DifPel as shown in FIG. 8(b), the numberof pixel difference calculations can be reduced for both filtered pixelcount controller 4 and filter coefficient controller 6. The filteredpixel count controller 4 and filter coefficient controller 6 cantherefore set the filtered pixel count and filter coefficient withoutreferencing reconstructed image Coef.

It will thus be apparent that the number of computations can be reducedby repeatedly using the value computed as pixel difference DifPel.

Embodiment 3

This embodiment of the invention describes an encoding apparatus and adecoding apparatus implementing the coding distortion removal methoddescribed in another embodiment of the invention.

FIG. 9 is a block diagram of the encoding apparatus.

Motion detection unit 30 compares reference image Ref1 and referenceimage Ref2 output respectively from first memory 38 and second memory 40with image signal Vin, and detects motion vector MV, that is, the amountof motion in image signal Vin relative to the reference image. It shouldbe noted that information indicating whether prediction error will beless by referencing reference image Ref1 or reference image Ref2 is alsoincluded in the motion vector MV and reported to motion compensationunit 32. The motion compensation unit 32 extracts the image at theposition indicated by motion vector MV from reference image Ref1 orreference image Ref2, and outputs it as motion compensated image MCpel.

Subtracter 42 obtains the difference of image signal Vin and motioncompensated image MCpel, and outputs to cosine transform unit (DCT) 46.Cosine transform unit 46 computes the DCT and quantizes the inputdifference, and outputs frequency component FCoef. Zigzag scanner 48outputs frequency code component DCoef reordering the sequence offrequency component FCoef, and variable length coding unit 50 variablelength codes frequency code component Dcoef to output encoded signalStr.

The output of the DCT unit (cosine transform unit) 46 is also input toinverse DCT unit (reverse cosine transform unit) 44. Frequency componentFCoef and motion compensated image MCpel output from motion compensationunit 32 are merged by synthesizer 34, and merged image Coef is output.The merged image Coef is stored as is to first memory 38, and is alsoprocessed by deblocking filter 36 and the decoded image signal Vout fromwhich coding distortion has been removed is stored to second memory 40.

FIG. 10 is a block diagram of the decoding apparatus. This decodingapparatus correctly decodes the encoded signal Str encoded by theencoding apparatus shown in the block diagram in FIG. 9. Parts in FIG.10 that operate the same as the corresponding parts in FIG. 32 or FIG.33 are identified by like reference numeral, and further thereofdescription is omitted here. The inverse DCT unit (reverse cosinetransform unit) 56 dequantizes frequency component FCoef and computesthe inverse DCT to output difference image DifCoef. The adder 58 addsdifference image DifCoef and motion compensated image MCpel to obtainreconstructed image Coef. Reconstructed image Coef is stored to firstmemory 64, and decoded image signal Vout obtained by deblocking filter62 removing coding distortion from reconstructed image Coef is stored tosecond memory 66.

As a result of this operation an image from which coding distortion isnot removed is stored to first memory 38 and first memory 64, and animage from which coding distortion is removed is stored to second memory40 and second memory 66. The coding distortion removal process does notalways remove only coding distortion, and it is possible that part ofthe actual image signal is also lost. The encoding apparatus shown inFIG. 9 is therefore configured so that the motion detection unit 30 canalways select the best output from both first memory 38 and secondmemory 40.

If part of the original image signal is lost by removing codingdistortion with the configuration of this embodiment, an appropriatereference image can be selected by referencing first memory 38. Anappropriate reference image can likewise be selected by the decodingapparatus shown in FIG. 10.

It should be noted that a DCT is used as the orthogonal transform inthis embodiment of the invention, but a Hadarnard transform or wavelettransform could be used.

Embodiment 4

FIG. 11 is a block diagram of a coding distortion removal unit accordingto a preferred embodiment of the invention, and corresponds to thedeblocking filter 62 shown in FIG. 1, for example. This codingdistortion removal unit is distinguished by determining the thresholdvalue for setting the filter. It should be noted that parts performingthe same operation as like parts in the coding distortion removal unitshown in FIG. 34 are identified by like reference numerals and furtherdescription thereof is omitted here.

Filter setting parameter decoder 22 decodes filter setting parametersignal FtrStr, and outputs filter parameter FtrPrm. This filter settingparameter signal FtrStr is not a threshold value, but is a parameter forsetting the threshold value. Filter parameter FtrPrm is equivalent to π,Ω, and ø in FIG. 5. By decoding and obtaining data optimizing theseparameters π, Ω, and ø for each picture from filter setting parametersignal FtrStr, coding distortion removal appropriate to the image isenabled.

FIG. 12 shows the structure of encoded signal Str in the codingdistortion removal method of the present invention. FIG. 12(a) is anencoded signal for one picture, and contains picture data PicDataholding the data for one picture, and picture header PicHdr common toall data in one picture. This to picture header PicHdr contains thefilter setting parameter signal FtrStr.

FIG. 12(b) shows the structure of picture data PicData. This picturedata PicData contains slice signal SliceStr, the encoded signal of aslice containing a group of plural block units.

FIG. 12(c) shows the structure of slice signal SliceStr, which containsslice data SliceData holding the data for one slice, and slice headerSliceHdr common to all data in the one slice. By writing filter settingparameter signal FtrStr to the slice header SliceHdr, an encoded signalreceived in slice data SliceData units can be correctly decoded.

If plural slice signals SliceStr are contained in picture data PicData,filter setting parameter signal FtrStr could be written to only some ofthe slice headers SliceHdr instead of writing filter setting parametersignal FtrStr to all slice headers SliceHdr. If the content of thefilter setting parameter signal FtrStr is common to each slice, andfilter setting parameter signal FtrStr is not written to the sliceheader SliceHdr as shown in FIG. 12(c), an increase in the number ofbits due to repeating the filter setting parameter signal FtrStr can besuppressed by substituting filter setting parameter signal FtrStr fromanother slice header SliceHdr.

If the encoded signal Str is transmitted in small data units such aspackets instead of as a single continuous bit stream, the header andnon-header parts can be separately transmitted. In this case the headerand data parts will not be in a single bit stream as shown in FIG. 12.However, even if the transmission sequence of the header and data partsis not continuous, the header for a particular data packet is simplytransmitted in another packet, and the concept is the same as the bitstream shown in FIG. 12 even though the transmission is not a single bitstream.

FIG. 13 is a block diagram of the encoding apparatus. Note that likeparts in FIG. 13 and FIG. 9 are identified by like reference numeralsand further description thereof is omitted here.

Memory 217 stores image signal Vin, that is, the image signal input forencoding. Image quality comparison unit 216 compares the encoding targetimage signal read from memory 217 with decoded image signal Vout. Thesize of the error obtained from the comparison done by image qualitycomparison unit 216 is stored together with the deblocking filterthreshold value for the decoded image to comparison memory 218. Theselection unit 219 selects as the optimum threshold value the thresholdvalue of the deblocking filter corresponding to the smallest errorstored in comparison memory 218. The selected optimum threshold value ismultiplexed as a related added bit stream to the bit stream of thecorresponding picture. Based on the optimum threshold value output byselection unit 219, threshold value control unit 215 generates acandidate threshold value for the deblocking filter of the next picture,advises the deblocking filter 36 and changes the threshold value of thecoding distortion removal process, and sends the threshold valuecurrently in use to the comparison memory 218.

FIG. 14 is a conceptual representation of the specific encodingapparatus shown in the block diagram in FIG. 13. In FIG. 14 the optimumthreshold value selection unit 226 performs the operations of the partsin FIG. 13 other than zigzag scanner 48, variable length coding unit 50,and threshold value appending unit 220, equivalent to the operation ofmemory 217, image quality comparison unit 216, comparison memory 218,selection unit 219, and threshold value control unit 215. The videoencoder 227 corresponds to the operation of the parts other than thememory 217, image quality comparison unit 216, comparison memory 218,selection unit 219, and threshold value control unit 215 in FIG. 13.Threshold value 228 is equivalent to the above optimum threshold value.

The optimum threshold value selection unit 226 selects an optimumthreshold value. This optimum threshold value is equivalent to the setof π, Ω, and ø values determined for each quantization parameter QP inFIG. 5. The selected optimum threshold value is stored to thresholdvalue memory 228 and applied to video encoder 227 as filter settingparameter signal FtrStr. The encoded filter setting parameter signalFtrStr is processed by the filter setting parameter decoder 22 shown inFIG. 11, for example, in the decoder.

It should be noted that the optimum threshold value could be stored inmemory in threshold value control unit 215 shown in FIG. 13, and thethreshold value data sent by threshold value control unit 215 tothreshold value appending unit 220.

An operation whereby filter setting parameter signal FtrStr isdetermined when removing coding distortion is described next. FIG. 15,FIG. 16, and FIG. 17 are flow charts showing the operation of theencoding apparatus described with FIG. 13 and FIG. 14.

FIG. 15 is a flow chart of an operation for measuring image quality.

The target frame target_frame is first set and the first picture output(step 229). The target frame target_frame is the picture used forderiving the threshold value.

The threshold value control unit 215 then sets a threshold value range(step 230), and the value at one end of this range is output fromthreshold value control unit 215 as the initial threshold value (step231).

Using this initial threshold value the deblocking filter 36 removescoding distortion, begins coding the picture for target frametarget_frame (step 232), and image quality comparison unit 216 thenmeasures the image quality of this first encoded picture and imagesignal Vin (step 233).

The result of this comparison is stored to comparison memory 218 (step234), and the current frame number current_frame is incremented (step235). That is, the picture being processed is changed from the firstpicture to the next picture, and the next picture is output to, forexample, optimum threshold value selection unit 226 and video encoder227 shown in FIG. 14 or memory 217, motion detection unit 30, andsubtracter 42 shown in FIG. 13.

Step 236 then determines if the current frame number current_frame hasreached the target frame target_frame. If it has not, steps 233 to 235repeat. The image quality of the input picture is measured by imagequality comparison unit 216, and the result is stored to comparisonmemory 218, If the current frame number current_frame equals thetarget_frame target_frame, control advances to step 237 and the currentframe number current_frame is reset to the first picture.

The threshold value control unit 215 then increments the threshold value(step 238A), that is, the threshold value is set to the next value. This“next value” is the value increased a specific increment from the firstvalue.

Whether all threshold values to the threshold value at the other end ofthe set range have been tested is then determined (step 238B). If allthreshold values have been tested, the process for determining theoptimum threshold value ends. If all threshold values have not beentested, control loops back to step 232 and the picture for target frametarget_frame is encoded.

Image quality can thus be measured by measuring the image quality forall target frames target_frame using one threshold value, thenincrementing the threshold value a specific amount, and then againmeasuring image quality for all target frames target_frame.

Referring next to the flow chart in FIG. 16, a method for measuringimage quality in one picture using all threshold values in a setthreshold value range, then advancing to the next picture and measuringimage quality using all threshold values in a set threshold value range,is described.

The target frame target_frame is first set and the first picture output(step 239). The current frame number current_frame is then initializedto 0 (step 240).

The threshold value control unit 215 then sets a threshold value range(step 241), and the threshold value is set to the deblocking filter 36(step 242).

The first picture is then encoded (processed for coding distortionremoval) using the initial threshold value (step 243), and the imagequality of the encoded picture is measured by image quality comparisonunit 216 (step 244).

The result output by image quality comparison unit 216 is stored tocomparison memory 218 (step 245), and the threshold value control unit215 increments the threshold value to the next value (step 246A).

Whether all threshold values have been tested is then determined (step246B). If all threshold values have not been tested, control loops backto step 242 and the image quality of the same picture is measured usinga different threshold value. If all threshold values have been tested,control advances to step 247.

The current frame number current_frame is then incremented in step 247.That is, the picture being processed is changed from the first pictureis (the first frame) to the second picture (the second frame), and thenext picture is output to, for example, optimum threshold valueselection unit 226 and video encoder 227 shown in FIG. 14 or memory 217,motion detection unit 30, and subtracter 42 shown in FIG. 13.

Step 248 then determines if the current frame number current_frame hasreached the target frame target_frame. If it has not, steps 241 to 247repeat. If current frame equals target_frame, the image qualitymeasurement process ends.

FIG. 17 is a flow chart of a method for selecting the optimum thresholdvalue based on the threshold value described in FIG. 15 or FIG. 16 andthe results of measuring image quality at that threshold value.

The selection unit 219 gets the image quality measurement results andcorresponding threshold value data in step 249 in FIG. 17.

The measurement results are then arranged in a specific order (step250).

The picture with the best image quality is then selected based onspecific conditions (step 251), and the threshold value for that pictureis selected as the optimum threshold value. These specific conditionscould be any one of or a combination of the following: a low S/N ratio,the smallest difference between the reconstructed image (the picturedeblocked at the threshold value) and the original picture (input imagesignal Vin), and the lowest mean square of the difference.

The selected optimum threshold value is then output as filter settingparameter signal FtrStr to, for example, video encoder 227 in FIG. 14(step 252).

The best threshold value can thus be selected using the method describedwith reference to FIG. 17.

As described above this preferred embodiment measures image quality forall threshold values in a specified range, gathers the image qualitymeasurement results, and selects the optimum threshold value from amongthe results. It is also possible to measure image quality in sequencefor all threshold values in a threshold value range, end image qualitymeasurement at the point a result with the best image quality isdetected, and select the threshold value producing that image qualityresult as the optimum threshold value. This method can reduce the numberof image quality measurements performed.

The coding distortion removal process for a given block compares thepixel values in that block with the pixel values in an adjacent block.The adjacent block in this case is a block for which the codingdistortion removal process has ended and pixel value correction hasended.

When removing coding distortion from block G in FIG. 18, for example,coding distortion could be removed by comparison with any of the fouradjacent blocks E, D, H, and M. However, by using a block for whichcoding distortion removal processing has already been completed, codingdistortion can be removed more accurately.

Coding distortion is preferably removed in linear sequence in thescanning order. That is, coding distortion is removed in the scanningdirection of the horizontal scan lines of the picture in horizontal scanline sequence.

In other words, referring to FIG. 18, the first scan line of blocks A,B, E, F is processed first for coding distortion removal, then the nextline of blocks C, D, G, H is processed, and so forth. Each block hasfour boundaries, but coding distortion removal processing is preferablyapplied using the adjacent blocks touching the top boundary and leftboundary.

In this case coding distortion removal processing is not applied toblock A because there is an adjacent block touching its top boundary orleft boundary.

There is similarly no adjacent block touching the top boundary of blockB, and deblocking is therefore applied using block A, which is adjacentto the left boundary of block B.

Blocks E and D are respectively adjacent to the top and left boundariesof block G, and coding distortion is therefore removed from block Gusing blocks E and D while not using blocks H and M.

By thus removing coding distortion between a new block and adjacentblocks from which coding distortion has already been removed, and notreferencing adjacent blocks that have not been processed for codingdistortion, coding distortion can be removed more accurately.

Embodiment 5

This embodiment first describes a case in which pixels are divided intogroups of multiple pixels each, such as groups of four pixels in onecolumn, groups are then paired, and coding distortion removal is appliedto group pairs. A coding distortion removal process as used in thisembodiment refers to both or either determining whether to applydeblocking to an area on both sides of a block boundary, and thedeblocking operation itself. A block could be a 4×4 block of 16 pixelsthat is the smallest coding unit, or any of the blocks to which motioncompensation is applied as described above with reference to FIG. 3.

As shown in FIG. 19 the four pixels in one group are a group of fourpixels arranged in line with the block boundary. Four such groups areshown in FIGS. 19, r1, r2, r3, and r4. Data from these four groups r1,r2, r3, and r4 can be stored to four registers (SIMD registers, forexample). Groups r1, r2 and groups r3, r4 are symmetrically located onleft and right sides of the block boundary. Pixel values in group r1 arecompared with pixel values in group r2, and coding distortion removalprocessing is applied using the resulting differences.

More specifically, difference 1 between the top pixel in group r1 andthe top pixel in group r2, difference 2 between the second to the toppixel in group r1 and the second to the top pixel in group r2,difference 3 between the second to bottom pixel in group r1 and thesecond to bottom pixel in group r2, and difference 4 between the bottompixel in group r1 and the bottom pixel in group r2 are obtained. Theaverage of difference 1, difference 2, difference 3, and difference 4,or the sum of the absolute values of difference 1, difference 2,difference 3, and difference 4, is used as a representative difference,and this representative difference is compared with a specific thresholdvalue. Other methods are also possible. Because these operations areperformed on units of four pixels in the same groups, parallelprocessing can be used for significantly faster throughput compared withprocessing each pixel at a time.

While comparison using just group r1 and group r2 is described above, ifgreater accuracy is required the luminance of pixels in group r3 can becompared with pixel luminance values from group r4, and therepresentative differences from the comparison of groups r1 and r2 canbe added to or averaged with the representative differences from groupsr3 and r4 to remove coding distortion.

The operation described above applies to vertical block boundaries, butthe same essential operation can be applied to horizontal boundaries bysimply assembling horizontal groups of four pixels along the horizontalboundaries.

FIG. 20(a) and (b) show cases in which the scan lines are interlaced onscreen. An interlaced picture is a picture in which one frame consistsof two fields presented at different times. Coding and decoding aninterlaced picture can be accomplished by processing one frame as aframe, as two fields, or by frame structure or field structure blocks inone frame. In FIG. 20 the small gray squares denote odd-line pixels, andthe small white squares denote even-line pixels. The gray pixels of theodd lines thus form one field of a frame and the white pixels on theeven lines form the other field of the same frame.

In an interlaced picture signal one frame consists of two fields (aneven field and an odd field) at different time instants. In a stillpicture the pixel values do not change with time, and the correlationbetween vertically adjacent lines in a frame is stronger than thecorrelation between vertically adjacent lines in a field. In a movingpicture, however, the picture changes greatly with time, pixel valuescan thus differ greatly in two fields, and the correlation betweenvertically adjacent lines in a field is stronger than the correlationbetween vertically adjacent lines in a frame. It is therefore moreefficient to process still pictures by frame and moving pictures byfield.

In an interlaced picture (1) all blocks could be frame structure blocks(the frame structure is described further below), (2) all blocks couldbe field structure blocks (the field structure is described furtherbelow), or (3) the picture could contain both frame structure and fieldstructure blocks.

If the picture contains all frame structure blocks (1), all deblockingis applied by frame structure unit. If the picture contains all fieldstructure blocks (2), all deblocking is applied by field structure unit.If the picture contains both frame structure and field structure blocks(3), deblocking is applied while adaptively converting from fieldstructure to frame structure or from frame structure to field structure.These operations are described more specifically below.

Interlaced pictures that are still images or contain little motion areprocessed by frame units consisting of odd fields and even fields asshown in FIG. 20(a) (referred to herein as a “frame structure”). In aframe structure, as shown on the right side in FIG. 20(a), a block of 16pixels contains both odd-line pixels and even-line pixels. The codingdistortion removal process is applied between blocks with a framestructure. That is, as described with reference to FIG. 8(b), codingdistortion removal processing is applied to the block boundaries.

Interlaced pictures with much motion are processed by field unitseparated into odd fields and even fields as shown in FIG. 20(b)(referred to herein as a “field structure”). As shown on the right sidein FIG. 20(b), the picture is separated into odd fields of odd-lines andeven fields of even-lines; odd fields contain blocks of odd-lines, andeven fields contain blocks of even-lines. The coding distortion removalprocess is applied only between field structure blocks of only odd-linesor field structure blocks of only even-lines.

FIG. 21(a) shows a case in which part of the interlaced image consistsof frame structure blocks and another part consists of field structureblocks. Preferably, the moving picture part of the image contains thefield structure blocks and the still picture part contains the framestructure blocks. The smallest unit formed by a field structure or framestructure is the macroblock, i.e., the largest unit to which DCT orother orthogonal transform or motion compensation is applied (orsuper-macroblocks of plural macroblocks). It is assumed below that therectangle containing the car in FIG. 21(a) contains field structureblocks, and the rest of the picture contains frame structure blocks.

How coding distortion removal is applied to the boundary between thefield structure part and the frame structure part is described next.

Referring to FIG. 21(b), the blocks in columns C1, C2, C3, and C4 belongto the image area containing the car and thus have a field structurebecause of the motion in this image area. The blocks in columns C5, C6,C7, and C8 belong to the area where the car is not, that is, the stillpicture area, and thus have an efficient frame structure. Note that inthis example the macroblocks have 16 pixels per side and the blocks have4 pixels per side. Columns C4 and C5 are shown apart in FIG. 21(b) butare actually adjacent in the picture. Coding distortion removal as shownin FIG. 8(b) is applied to the block boundary between columns C3 and C4and the block boundary between columns C5 and C6.

To process the block boundary between columns C4 and C5 the framestructure blocks in column C5 are first converted to field structureblocks as shown in FIG. 21(c). This is done by, for example, convertingthe odd-line pixels in column C5 shown in FIG. 21(b) to a block of graypixels in column C5 as shown in FIG. 21(c), and converting the even linepixels in column C5 shown in FIG. 21(b) to a block of white pixels incolumn C5 as shown in FIG. 21(c). Coding distortion at the blockboundary between columns C4 and C5 is then removed as shown in FIG.8(b).

Frame structure blocks are thus converted to field structure blocksbecause the vertical correlation between pixels will be lost if fieldstructure blocks are converted to frame structure blocks when there ismovement in the picture, and unnatural degradation occurs if the codingdistortion removal process is applied between vertically adjacentblocks. On the other hand, while suppression of coding error in highfrequency components in the vertical direction is reduced if framestructure blocks are converted to field structure blocks in stillpictures, the vertical correlation between pixels is not lost andunnatural image quality degradation does not occur easily.

Frame structure blocks are converted to field structure blocks to reducethe amount of processing (only converting frames to fields) in the aboveexample. However, if the number of operations is not of concern, analternative method can be used that converts frames to fields and fieldto frames, and thus increases the number of operations compared with theprevious example because of the additional processing required toconvert fields to frames. More specifically, whether the target pixelsfor coding distortion removal (i.e., the current pixel for which thepixel value is to be changed by deblocking) are in a frame structureblock or a field structure block is first determined. If the targetpixels for coding distortion removal are in a field structure block,frame structure blocks are converted to field structure blocks (i.e.,the block type of the target pixel), and if the target pixels for codingdistortion removal processing are in a frame structure block, fieldstructure blocks are converted to frame structure blocks (i.e., theblock type of the target pixel).

Operation when frame structures and field structures are mixed isdescribed next with reference to the flow chart in FIG. 22.

A frame in an interlaced image signal stream consists of two fieldsscanned at different time instants. A frame can therefore be frameencoded by combining the two fields into a single coding unit (framestructure coding), or it can be field encoded with the two fields codedand handled separately (field structure coding). These coding methodscan also be grouped into the following two categories, fixed coding andadaptive coding. With fixed coding the entire picture is switchedbetween either frame coding or field coding. With adaptive coding thepicture is divided into a number of blocks and each block is eitherframe encoded or field encoded.

Fixed coding further includes frame-fixed coding applied to framestructure blocks, and field-fixed coding applied to field structureblocks. With fixed coding the interlaced video sequence is alwaysencoded with either frame encoding or field encoding regardless of thecontent.

With adaptive coding, however, frame encoding or field encoding can beadaptively selected based on the content, the picture, or coding blockunit in the picture. These in-picture coding blocks can be as small asthe macroblock. With adaptive coding individual macroblocks cantherefore be coded using either frame encoding or field encoding.Macroblocks are used as the coding unit below.

Frame encoded blocks, that is, blocks with a frame structure, can beprocessed for coding distortion removal using the same technique appliedto non-interlaced video.

With field encoded blocks, that is, blocks with a field structure, thefields are separated into even fields and odd fields, each field ishandled as a separate picture, and deblocking is therefore applied toeach field.

Referring to the flow chart in FIG. 22, whether the target block isfield encoded or frame encoded is decided first (step 63). If the blockis field encoded, steps 64 to 69 are run. If the block is frame encoded,steps 70 to 72 run.

Steps 64 to 66 process even field structure blocks, and steps 67 to 69process odd field structure blocks. Steps 64 to 66 remove codingdistortion between white pixels at the boundary between columns C3 andC4 in FIG. 21(b), and steps 67 to 69 remove coding distortion betweengray pixels at the boundary between columns C3 and 04 in FIG. 21(b).

More specifically, pixel luminance is compared in step 64 to determinewhether coding distortion removal is needed. The number of pixels to befiltered is then determined in step 65. Coding distortion is thenremoved in the field mode in step 66.

Steps 67, 68, and 69 perform the same operations as steps 64, 65, and66, respectively.

Steps 70 to 72 process frame structure blocks to remove codingdistortion at the boundary between columns C5 and C6 in FIG. 21(b). Morespecifically, pixel luminance is compared in step 70 to determinewhether coding distortion removal is needed. The number of pixels to befiltered is then determined in step 71. Coding distortion is thenremoved in the frame mode in step 72.

Whether all blocks have been processed is determined in step 73, and ifthey have operation ends.

FIG. 23 shows an alternative method in which steps 64 and 67 in FIG. 22are combined into a single step. More specifically, whether it isnecessary to remove coding distortion from both even field blocks andodd field blocks is determined, and deblocking is applied to both evenand odd field blocks if it is needed. This simplifies the codingdistortion removal process.

FIG. 24 shows a further alternative method in which steps 65 and 68 inFIG. 23 are combined into a single operation determining the number ofpixels in both the even field blocks and odd field blocks to bedeblocked. Coding distortion removal is then applied to both even andodd field blocks based on the result. This method further simplifiescoding distortion removal.

FIG. 25 is a flow chart of a process used when frame encoded blocks andfield encoded blocks are mixed in a single picture, and the blockboundary is between a frame structure block and a field structure block.

Step 95 first determines if the boundary line between the blocks beingprocessed for coding distortion removal is a specific boundary line,that is, if a frame structure block is on one side of the line and afield structure block is on the other side. This is comparable todetermining if the line is between columns C4 and C5 in FIG. 21(b). Ifit is (step 95 returns yes), control advances to step 96.

The frame structure block on one side of the boundary is then convertedto a field structure block (step 96). This conversion is comparable toconverting a block in column C5 in FIG. 21(b) to a block in column C5 inFIG. 21(c). The converted block is referred to below as a “conversionblock.”

Whether coding distortion removal is needed between the conversion blockand the field structure block on the other side of the boundary is thendetermined (step 97). This is comparable to deciding whether deblockingis needed at the boundary between columns C4 and C5 in FIG. 21(c). If itis needed, control advances to step 98.

The number of pixels to filter is then determined (step 98), and codingdistortion is removed in the field mode (step 99).

FIG. 25 shows a method whereby frame structure blocks are converted tofield structure blocks and coding distortion is removed from the fieldswhen adaptively coded frame structure and field structure blocks areadjacent, but it is conversely possible to convert field structureblocks to frame structure blocks, and remove coding distortion on aframe basis.

An advantage of removing coding distortion on a field basis as shown inFIG. 25 is that operation is resistant to unnatural image qualitydegradation because coding distortion is removed using only pixels atthe same time instant even in image signals with rapid motion. On theother hand, because the correlation between pixels in the verticaldirection is stronger in frames than fields in image signals with littlemotion, deblocking on a frame basis results in less degradation of highfrequency components than does deblocking on a field basis. Both methodsthus have advantages, and the equipment manufacturer could select thepreferable method or means could be provided so that the user can selectthe desired method.

Coding distortion removal could also be applied by picture unit (frameor field) instead of by block unit with adaptive coding. The deblockingfilter can be simplified by providing one field mode or frame modedeblocking filter for processing picture units. The filter could befixed in the field mode or frame mode, or it could switch on a picturebasis. If the filter switches on a picture basis, the coding apparatuscan determine the appropriate mode, and an identification signaldenoting whether the deblocking filter of the decoding apparatus shouldoperate in the field mode or frame mode can be added to the code streamheader and transmitted to the decoder.

Furthermore, when field or frame mode operation can switch on a blockunit basis and deblocking and switching on a field basis is prohibited(by setting a picture parameter to prohibit switching in the picture,for example), coding distortion can be removed by frame units.

It should be noted that the deblocking filter in the first to fifthembodiments described above can be used as a post filter as shown inFIG. 32 or an in-loop filter as shown in FIG. 33.

By storing the data from before the deblocking operation to memory 64,an image from which block distortion has not been removed is referencedas the predictive picture when used as an in-loop filter, and there isslightly more degradation of the encoded image quality compared withusing a deblocked picture as the predictive picture.

On the other hand, because the result of removing coding distortion isnot used as the reference image when used as a post filter, the decodedimage will not be greatly degraded regardless of the type of deblockingfilter 62 used. For example, a simple filter performing the fewestoperations could be used as the deblocking filter 62 in a cell phone, adevice for which low power consumption is a priority, while a highprecision, high image quality filter could be used as the deblockingfilter 62 in a stationary entertainment system for which image qualityis the top priority.

Embodiment 6

By recording a program implementing the steps of the coding distortionremoval method, coding method, and decoding method described in thepreceding embodiments to a floppy disk or other computer-readable datarecording medium, the processes described in the above embodiments canbe easily executed on an independent computer system.

FIG. 26 shows a computer system as a further embodiment of the inventionachieved using a data recording medium (a floppy disk in this example)storing the coding distortion removal method, coding method, anddecoding method described in the first to fifth embodiments above.

FIG. 26(b) shows a floppy disk as seen from the front, a section view ofthe same, and the actual disk medium, and FIG. 26(a) shows the physicalformat of a typical floppy disk recording medium. The floppy disk FD ishoused inside a case F. A plurality of concentric tracks Tr are formedfrom the outside circumference to the inside circumference on the disksurface, and the tracks are divided in the angular direction into 16sectors Se. A floppy disk FD storing the above program according to thepresent invention thus has the coding distortion removal method, codingmethod, and decoding method of the invention recorded ascomputer-executable programs to specifically allocated areas on thefloppy disk FD.

FIG. 26(c) shows an apparatus for recording and, reading these programsusing this floppy disk FD. To record these programs to the floppy diskFD, the computer system Cs writes the coding distortion removal method,coding method, and decoding method as the programs by means of a floppydisk drive FDD. To execute the coding distortion removal method, codingmethod, and decoding method on the computer system from the programsstored to the floppy disk FD, the programs are read from the floppy diskFD by the floppy disk drive and transferred to the computer system.

It should be noted that while a floppy disk is described above as thedata recording medium, an optical disc or other type ofcomputer-readable medium could be used, including CD-ROM discs, memorycards, ROM cassettes, or any other medium capable of similarly recordingthe programs.

A system applying the video coding method and video decoding methodaccording to the above embodiments is described next.

FIG. 27 is a schematic diagram showing the overall configuration of acontent supply system ex100 for providing a content distributionservice. The service area of this communication system is divided intocells of a desired size, and a base station ex107 to ex110 (stationarywireless station) is installed in each cell.

This content supply system ex100 has numerous individual devices such ascomputer ex111, PDA (Personal Digital Assistant) ex112, camera ex113,cell phone ex114, and a cell phone with a camera ex115 connected to theInternet ex101, for example, by means of Internet service providerex102, telephone network ex104, and base stations ex107 to ex110.

This content supply system ex100 shall not be limited to theconfiguration shown in FIG. 27, however, and the desired devices couldbe selectively connected. The individual devices could also be connecteddirectly to telephone network ex104 without passing through the fixedbase stations ex107 to ex110.

Camera ex113 is a digital video camera or other device capable ofcapturing video images. The cell phone could use any of variousprotocols, including PDC (Personal Digital Communications), CDMA (codedivision multiple access), W-CDMA (wideband code division multipleaccess), GSM (Global System for Mobile Communications), and PHS(Personal Handyphone System).

The camera ex113 can connect via a base station ex109 and telephonenetwork ex104 to a streaming server ex103, which can stream livebroadcasts of encoded content sent by a user using camera ex113. Thecontent received from the camera ex113 can be encoded by the cameraex113 or by the server. Video data captured with a camera ex116 can alsobe sent via computer ex111 to the streaming server ex103. This cameraexile is a digital camera or other device capable of capturing bothstill pictures and video. The video data received from the camera ex116can be encoded by the camera ex116 or by the computer ex111. In eithercase the video data is processed by LSI device ex117 in the computerex111 or camera ex116. The software for video coding and decoding can bestored to any computer-readable data recording medium (such as a CD-ROMdisc, floppy disk, or hard disk drive) that the computer ex111 canaccess.

Video data could also be sent by a cell phone with a camera ex115. Thevideo data in this case is encoded by an LSI device in the cell phonewith a camera ex115.

With this content supply system ex100, content (such as a live recordingof a concert) recorded by the user using camera ex113, camera ex116, orother device is coded as described in the above embodiments of theinvention and sent to the streaming server ex103. The streaming serverex103 then streams the content data out to clients requesting the data.The clients could be any device capable of decoding the encoded content,including computer ex111, PDA ex112, camera ex113, and cell phone ex114.This content supply system ex100 thus enables clients to receive andreproduce encoded content data, enables the clients to receive, decode,and play back content in real-time, and is thus a system enablingpersonal broadcasting,

The video coding apparatus and video decoding apparatus of the presentinvention described in the above embodiments can be used for coding anddecoding by the individual devices in this content supply system ex100.

A cell phone used in this content supply system ex100 is described nextby way of example.

FIG. 28 shows a cell phone ex115 using the video encoding method andvideo decoding method described above according to the presentinvention. As shown in FIG. 28 this cell phone with a camera ex115 hasan antenna ex201 for exchanging RF signals with a base station ex110; acamera ex203 such as a CCD camera for capturing video and stillpictures; a display unit ex202 such as an LCD for displaying imagescaptured by the camera ex203 or images received by antenna ex201 andthen decoded; an operating panel with a keypad ex204 and other controls;an audio output unit such as a speaker ex208 for outputting audio; amicrophone ex205 or other type of audio input device; recording mediumex207 for storing encoded or decoded data such as video or still imagedata captured by the camera ex203, received e-mail, or other video orstill picture data; and a slot ex206 for loading recording medium ex207into the cell phone ex115. The recording medium ex207 could be an SDCard or other type of flash memory device such as an EEPROM(electrically erasable and programmable read only memory) housed in aplastic case.

This cell phone ex115 is further described with reference to FIG. 29.Connected to the main controller ex311 for systematically controllingeach part of the cell phone ex115 including the display unit ex202 andkeypad ex204 via synchronization bus ex313 are a power supply circuitex310, operating input controller ex304, image encoding unit ex312,camera interface ex303, LCD controller ex302, image decoding unit ex309,multiplexer/demultiplexer ex308, reading/writing unit ex307,modulator/demodulator unit ex306, and audio processing unit ex305.

When the user sets the end and power buttons to the on position, powersupply circuit ex310 supplies power from a battery pack to each part ofthe cell phone ex115 and thus sets the digital cell phone ex115 withcamera to the operating mode.

Controlled by the main controller ex311, which typically includes a CPU,ROM, and RAM, cell phone ex115 converts the audio signals picked up bythe microphone ex205 when in the talk mode to digital audio data bymeans of audio processing unit ex305. The modulator/demodulator unitex306 then spectrum-spreads audio processing unit ex305 output, and thecommunication circuit ex301 applies DIA conversion and frequencyconversion processing, and then outputs through antenna ex201. When inthe talk mode the cell phone ex115 amplifies signals received throughthe antenna ex201 and applies frequency conversion and ND processing,the modulator/demodulator unit ex306 despreads the signal, the audioprocessing unit ex305 then converts the as despread signal to an analogaudio signal, and outputs the analog audio signal from speaker ex208.

If e-mail is sent when in the data communication mode, the text data ofthe e-mail message is input using the keypad ex204, and sent throughoperating input controller ex304 to main controller ex311. The maincontroller ex311 then spectrum-spreads the text data usingmodulator/demodulator unit ex306, D/A converts and frequency conversionprocesses the signal using communication circuit ex301, and thentransmits from antenna ex201 to base station ex110.

To transmit image data when in the data communication mode, image datacaptured with the camera ex203 is supplied through camera interfaceex303 to image encoding unit ex312. If the image data is nottransmitted, image data captured with the camera ex203 can be displayeddirectly on the display unit ex202 by way of camera interface ex303 andLCD controller ex302.

The image encoding unit ex312 has the configuration of an image encodingapparatus according to the present invention. It converts image datasupplied from camera ex203 to encoded image data by compression codingusing the coding method used in the image encoding apparatus describedin the preceding embodiments, and outputs the encoded image data to themultiplexer/demultiplexer ex308. Audio captured by the microphone ex205of cell phone ex115 while recording with the camera ex203 is also sentto the multiplexer/demultiplexer ex308 as digital audio data by theaudio processing unit ex305.

The multiplexer/demultiplexer ex308 multiplexes the coded picture datasupplied from image encoding unit ex312 with the audio data suppliedfrom audio processing unit ex305. The resulting multiplexed data is thenspectrum-spread by modulator/demodulator unit ex306, D/A conversion andfrequency conversion are applied by the communication circuit ex301, andthe signal is then transmitted from antenna ex201.

If data from a video file accessed from a web site on the Internet whenin the data communication mode is received, the signal received from thebase station ex110 via antenna ex201 is despread bymodulator/demodulator unit ex306, and the resulting multiplexed data issent to the multiplexer/demultiplexer ex308.

To decode the multiplexed data received through antenna ex201,multiplexer/demultiplexer ex308 demultiplexes the multiplexed data toseparate the encoded video data bitstream and the encoded audio databitstream. The encoded video data bitstream is then supplied to theimage decoding unit ex309 and the encoded audio data bitstream issupplied to the audio processing unit ex305 by way of synchronizationbus ex313.

The image decoding unit ex309 has the same configuration as the imagedecoding apparatus described in the above embodiments. It producesreconstructed video data by decoding an encoded video data bit streamusing a decoding method corresponding to the coding method describedabove, and supplies the decoded video data through LCD controller ex302on display unit ex202. Video data in a video file accessed from a webpage on the Internet can thus be displayed. The audio processing unitex305 also converts the audio data to an analog audio signal at the sametime, and supplies the result to the speaker ex208. Audio data containedin a video file accessed from a web site on the Internet can thus alsobe reproduced from the speaker.

The communication system of the present invention shall not be limitedto the above configuration. This system could, for example, be adaptedto a digital broadcasting system as shown in FIG. 30 using the imageencoding apparatus and/or the image decoding apparatus of the presentinvention to access digital broadcasts transmitted via satellite orterrestrial networks.

More specifically, broadcast station ex409 transmits an encoded videodata bit stream via radio waves to a communication or broadcastsatellite ex410. The broadcast satellite ex410 receiving thistransmission transmits the broadcast signal, which is received by anantenna ex406 in a home, for example, with a satellite broadcastreceiver. The encoded bit stream is then decoded and reconstructed bythe television receiver ex401, set-top box (STB) ex407, or other device.

The video decoding apparatus of the present invention can also beimplemented in a playback device ex403 for reading and decoding anencoded bit stream recorded to a recording medium such as a CD, DVD, orother storage medium ex402. In this case the reconstructed video signalis presented on a monitor ex404, for example.

The image decoding apparatus of the invention could also be built in toa set-top box ex407 connected to a satellite or terrestrial broadcastantenna ex406 or to a cable antenna ex405 for cable television access.Output from this set-top box ex407 could also be presented on atelevision monitor ex408.

The image decoding apparatus could alternatively be built in to thetelevision instead of the set-top box.

Signals could also be received from satellite ex410 or base stationex107 by an automobile ex412 having an appropriate antenna ex411, andthe decoded video could be presented on the display of a car navigationsystem ex413 in the automobile ex412.

A video signal could also be coded by a video encoding apparatusaccording to an embodiment of the present invention and recorded to adata recording medium. More specifically, a DVD recorder could recordthe image signal to a DVD disc ex421, or a hard disk recorder ex420could record the image signal. The video signal could furtheralternatively be recorded to an SD Card ex422. If the recorder ex420 hasa video decoding apparatus according to the present invention, it couldalso play back and present on monitor ex408 video signals recorded toDVD disc ex421, SD Card ex422, or other storage medium.

It should be noted that the car navigation system ex413 can beconfigured without the camera ex203, camera interface ex303, and imageencoding unit ex312 shown in FIG. 29. This also applies to the computerex111 and television (receiver) ex401, for example.

The cell phone ex114 or other terminal could be a transceiver terminalhaving both the above-described encoder and decoder, or it could be atransmission terminal having only the encoder, or a reception terminalhaving only the decoder.

It will also be obvious that the encoding apparatus an decodingapparatus of the present invention shall not be limited to theconfigurations described in the above first to sixth embodiments, andcan be varied in many ways.

The video encoding method and video decoding method described in theabove embodiments can thus be used in any of the devices and systemsdescribed above, thereby achieving the effects of these embodiments.

The coding distortion removal method of the present invention thusprovides a coding distortion removal method with a simple process, acoding distortion removal method with little likelihood of reducing theimage quality of the image signal due to removing coding distortion, anda coding method and decoding method that can reduce the likelihood ofdegrading the image quality of the image signal as a result of removingcoding distortion. The present invention therefore has great practicalvalue.

Although the present invention has been described in connection with thepreferred embodiments thereof with reference to the accompanyingdrawings, it is to be noted that various changes and modifications willbe apparent to those skilled in the art. Such changes and modificationsare to be understood as included within the scope of the presentinvention as defined by the appended claims, unless they departtherefrom.

What is claimed is:
 1. A picture coding method for coding an inputpicture, the picture coding method comprising: dividing the inputpicture into a plurality of blocks; coding the input picture on a blockbasis to obtain coded data of each block; decoding the coded data ofeach block on a block basis to obtain a reconstructed picture having aplurality of blocks; removing coding distortion in an area disposed onboth sides of a block boundary between a first block and an adjacentsecond block in the reconstructed picture having the plurality ofblocks, each block being adaptively decoded either as a field structureblock, comprising only even field pixels or comprising only odd fieldpixels, or a frame structure block, comprising odd field pixels and evenfield pixels; and storing the reconstructed picture, for which codingdistortion is removed, as a reference picture, wherein said removingcoding distortion further includes: detecting whether the block boundarybetween the first block and the adjacent second block is a blockboundary between a field structure block and a frame structure block;and performing a coding distortion removal process on the first blockand the adjacent second block, wherein the coding distortion removalprocess includes determining whether coding distortion removal isneeded, determining the number of pixels to be processed, and removingcoding distortion, wherein the removing of coding distortion includesperforming a coding distortion removal process on even field pixels ofthe first block and separately performing the coding distortion removalprocess on odd field pixels of the first block when it is detected thatthe block boundary between the first block and the adjacent second blockis a block boundary between a field structure block and a framestructure block, and the first block is a frame structure block and theadjacent second block is a field structure block, wherein the firstblock and the adjacent second block are blocks with 4 pixels×4 pixels,and the block boundary between the first block and the adjacent secondblock is a horizontal block boundary between a macroblock including thefirst block and a macroblock including the adjacent second block.
 2. Apicture coding apparatus that codes an input picture, the picture codingapparatus comprising: a dividing device operable to divide the inputpicture into a plurality of blocks; a coding device operable to code theinput picture on a block basis to obtain coded data of each block; adecoding device operable to decode the coded data of each block on ablock basis to obtain a reconstructed picture having a plurality ofblocks; a coding distortion removal device operable to remove codingdistortion in an area disposed on both sides of a block boundary betweena first block and an adjacent second block in the reconstructed picturehaving the plurality of blocks, each block being adaptively decodedeither as a field structure block, comprising only even field pixels orcomprising only odd field pixels, or a frame structure block, comprisingodd field pixels and even field pixels; and a storing device operable tostore the reconstructed picture, for which coding distortion is removed,as a reference picture, wherein said coding distortion removal device isfurther operable: to detect whether the block boundary between the firstblock and the adjacent second block is a block boundary between a fieldstructure block and a frame structure block; and to perform a codingdistortion removal process on the first block and the adjacent secondblock, wherein the coding distortion removal process includesdetermining whether coding distortion removal is needed, determining thenumber of pixels to be processed, and removing coding distortion,wherein the removing of coding distortion includes performing a codingdistortion removal process on even field pixels of the first block andseparately performing the coding distortion removal process on odd fieldpixels of the first block when it is detected that the block boundarybetween the first block and the adjacent second block is a blockboundary between a field structure block and a frame structure block,and the first block is a frame structure block and the adjacent secondblock is a field structure block, wherein the first block and theadjacent second block are blocks with 4 pixels×4 pixels, and the blockboundary between the first block and the adjacent second block is ahorizontal block boundary between a macroblock including the first blockand a macroblock including the adjacent second block.